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RELIABILITY  OF  THE  U.S.  ARMY  AMBULATORY  CARE  DATA  BASE  (ACDB) 

STUDY:  METHODOLOGY  AND  CLINICAL  FINDINGS 

SUMMARY 

During  the  study  period,  January  1986  -  September  1987,  researchers 
collected  data  on  3.1  million  patient  encounters  (visits)  at  six 
Army  hospitals.  After  the  data  collection  phase  of  the  ACDB  study,  members 
of  the  study  team  conducted  a  comprehensive  5-month  review  to  determine  a 
data  reliability  score  for  each  participating  hospital  and  clinical 
specialty.  The  study  team  reviewed  over  9,000  randomly  selected  visits  with 
their  supporting  medical  records.  They  evaluated  variables  of  interest  with 
a  specially  developed  scoring  instrument  which  they  used  to  assign  numerical 
weights  for  the  selected  variables. 

Reliability  mean  scores  were  computed  for  each  hospital  and  clinic  for 
the  two  data  collection  phases.  Phase  One  (I)  data  were  collected  during 
the  period  January  1986  through  April  1987.  Phase  Two  (II)  was  accomplished 
from  May  through  September  1987  using  modified  versions  of  the  original  data 
collection  forms.  The  reliability  mean  score  for  the  Phase  I  data  was  8.57 
(9  was  the  maximum  score)  with  a  standard  deviation  of  1.27  (n=  7,589).  The 
reliability  mean  score  for  the  Phase  II  data  was  8.50  with  a  standard 
deviation  of  1.31  (n  =  1,426).  These  results  indicate  a  high  degree  of 
reliability  between  the  key  variables  on  the  ACDB  visit  forms  and  the 
corresponding,  official  patient  medical  record. 

INTRODUCTION 

HiS.tg.cy  iM  Purp.QSg 

Recognizing  the  requirement  for  an  ambulatory  care  data  base,  the  Army 
Medical  Department  began  planning  in  1984  for  a  multi-year  study  to 
establish  an  outpatient  data  base.  Based  on  the  results  of  a  6-month  pilot 


study  completed  at  Fox  Army  Community  Hospital,  Redstone  Arsenal,  Alabama 
(Misener  &  Gilbert,  1984),  the  ACDB  Study  was  formulated  to  collect  clinical 
data  from  patient  encounters  (visits).  During  a  21-month  period  from 
January  1986  to  September  1987,  over  3.1  million  patient  encounters  were 
recorded. 

This  report  examines  the  reliability  of  the  clinical  data  obtained  from 
the  six  participating  Army  hospitals  and  their  respective  clinical 
specialties.  A  quantitative  measure  of  the  reliability  of  the  data  was 
determined  to  be  a  prerequisite  to  the  subsequent  analysis  by  clinical 
specialty. 

Background 

The  development  of  medical  classification  systems  is  not  a  recent 
innovation  (e.g.,  International  Classification  of  Diseases,  1979).  However, 
the  application  of  such  systems  to  hospital  management  and  reimbursement 
mechanisms  is  fairly  recent.  Specifically,  Diagnostic  Related  Groups  (DRGs) 
were  developed  for  this  purpose  (Fetter,  Averill,  Lichtenstein  &  Freeman, 
1984).  The  quest  for  more  efficient  management  and  more  equitable 
reimbursement  systems  led  to  the  development  of  other  similar  methodologies. 
These  include  Ambulatory  Visit  Groups  (AVGs) ,  Resource  Utilization  Groups 
(RUGs),  Products  of  Ambulatory  Care  (PACs),  and  others  (Kelly,  Fillmore,  & 
Tenan,  1988).  Central  to  the  development  of  these  and  other  classification 
systems  is  the  accurate  measurement  of  care  provided.  This  accuracy  of 
measurement,  more  commonly  referred  to  as  reliability  of  data,  is  not 
identified  with  glamorous  research.  It  is,  however,  essential  for 
confidence  in  results  obtained  from  any  subsequent  work  (Richards,  Lurie, 
Rodgers,  &  Brook,  1988). 

In  order  to  determine  the  reliability  of  the  Army's  ACDB,  a 
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comprehensive  reliability  study  was  conducted  at  all  six  test  hospitals  and 
included  the  outpatient  specialties  which  were  part  of  the  study.  Hospitals 
participating  in  the  study  were  Brooke  Army  Medical  Center  (BAMC) ,  Fort  Sam 
Houston  (San  Antonio),  Texas;  Bayne-Jones  Army  Community  Hospital,  Fort  Polk 
(Leesville),  Louisiana;  Womack  Army  Community  Hospital,  Fort  Bragg 
(Fayetteville),  North  Carolina;  Fox  Army  Community  Hospital,  Redstone 
Arsenal  (Huntsville),  Alabama;  Blanchfield  Army  Community  Hospital,  Fort 
Campbell  (Clarksville,  Tennessee),  Kentucky;  and  Moncrief  Army  Community 
Hospital,  Fort  Jackson  (Columbia),  South  Carolina. 

RESULTS 

The  four  variables  selected  in  the  reliability  study  are  part  of  each 
patient's  record.  Unfortunately,  the  reliability  study  was  not  able  to 
specifically  review  on  a  one-to-one  basis  the  other  fifty-seven  variables 
(e.g.,  time  spent  with  patient,  number  of  prescriptions,  pathology  tests  and 
radiography)  used  in  the  study.  Study  researchers  contend  that  the 
reliability  of  the  variables  selected  has  the  potential  to  imply  a  similar 
trend  in  other  variables. 

Quantifiable  reliability  measures  for  each  participating  hospital  and 
clinical  specialty  were  derived  from  an  exhaustive  review  of  supporting 
patient  medical  records.  Since  data  were  collected  during  two  consecutive 
but  separate  periods,  the  reliability  measures  were  computed  for  each  phase. 

Analysis  of  variance  tests  revealed  statistically  significant 
differences  between  hospital  sites,  phases  of  data  collection,  and 
individual  participating  clinical  specialties  do  exist.  However,  no 
practical  significance  or  practical  implications  were  found  as  a  result  of 
these  statistical  differences.  Moreover,  some  of  the  statistical 
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differences  found  can  be  a  function  of  large  sample  sizes  (Welch  & 

Comer,  1988). 

As  a  result  of  this  extensive  study,  the  question  of  the  accuracy  of 
the  studied  variables  can  be  answered  without  hesistation.  These  data  are 
unquestionably  of  a  very  high  quality  and  on  a  par  with  the  best  of  any 
medical  data  collected  and  scrutinized  within  or  outside  the  Army  Medical 
Department. 
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